Repository navigation
ci: generate baseline benchmark projects with the baseline release generator - #921
Merged
Merged
Conversation
…ator The release performance comparison generated its benchmark projects with the candidate's generator and compiled them with the previous release, so any generator syntax newer than that release (such as FUNCTION descriptions) failed the baseline before timing. The baseline now runs projects written by its own generator: the release tag's source is extracted with git archive (or taken from --baseline-source) and its write_pristine_projects runs with this interpreter, with cwd and PYTHONPATH set to that source, through the checkout generator runner shared with the same-runner compile ratio job. The candidate keeps its own generator. Generation failures stop the comparison with the generator's stderr tail, and the report and evidence record the baseline's generator.
… runner noise The E2E compares one build with itself using a single run, so asserting a passed verdict depended on wall-time noise (a 1.31x wall ratio at 0.99x CPU failed CI). It now asserts the report structure, every command row, the per-side generator markers and the recorded evidence, and accepts either verdict as long as the exit code matches it. The production 1.25x limits are unchanged; the verdict unit tests gain fixed-timing cases for a wall-only regression at CPU parity and a wall-only slowdown just under the limit. Fixes CHI-481
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Release PR #918 is blocked: the release-performance comparison generated its benchmark projects with the current generator, which now writes
FUNCTION (description …), and compiled them with the previous release, which rejects that key. Any generator change that adds syntax breaks this job. Its self-comparison E2E was also flaky on wall-time noise (CHI-481).Changes
git archiveor--baseline-source. The candidate keeps its own generator. The report records which generator produced the baseline's projects.run_checkout_generatorhelper for this job and the same-runner compile ratio job.Verification
make test,make check-ci.